Benchmarking Scalable Predictive Uncertainty in Text Classification

This paper explores the question of how predictive uncertainty methods perform in practice Natural Language Processing, specifically multi-class and multi-label text classification. We conduct benchmarking experiments with 1-D convolutional neural networks pre-trained transformers on six real-world classification datasets which we empirically investigate why popular scalable estimation strategies ( Monte-Carlo Dropout , xmlns:xlink="http://www.w3.org/1999/xlink">Deep Ensemble ) notable extensions xmlns:xlink="http://www.w3.org/1999/xlink">Heteroscedastic xmlns:xlink="http://www.w3.org/1999/xlink">Concrete underestimate uncertainty. motivate that benefits from combining posterior approximation procedures, linking it to recent research ensembles variational Bayesian navigate loss landscape. find our proposed method combination by analysis in- domain calibration, cross-domain classification, novel class robustness, demonstrates superior performance, even at a smaller ensemble size. Our results corroborate importance fine-tuning dropout rate task hand, individually as an impacts model robustness. observe ablation severely underperform novelty detection, limiting applicability transfer learning when distribution shift classes can be expected.